MedEval-six test collections in one

Information retrieval is a field of research reaching from computer and information science to lingusitics. As a linguist in the information retrieval field, I leave the quest for effective search engines and evaluation models to others, and focus on language aspects. Words, and parts of words such as compound constituents, which are successful in queries, what features do they have in common? Does the domain of search terms have impact in a domain specific environment? Can search terms with certain features help users of different categories find documents suited for them? This paper describes the making of an information retrieval test collection which made it possible to study these questions. The test collection will be used to Evaluate search strategies to retrieve Medical documents, hence the name. To study language aspects of information retrieval a new test collection was called for, a collection which was domain specific, which regarded user groups, and which had double indexes Table 1: The genres of the MedEval document sources. (D. Kokkinakis, p.c.) Type of source Number of Percent of Number Percent documents documents of tokens of tokens Journals and periodicals 8 453 20.0 5.3 million 34.6 Specialized sites 14 631 34.6 2.9 million 19.1 Pharmaceutical companies 9 200 21.8 2.3 million 14.8 Faculties, institutes, hospitals and government 2 955 7.0 2.0 million 13.3 Health-care communication companies 4 036 9.6 1.7 million 11.3 Media (TV, daily newspapers) 2 980 7.1 1.0 million 6.9 Total 42 255 100.1 15.2 million 100 for split and unsplit compounds. Since there was no such collection we built MedEval, a Swedish medical test collection, with documents marked for target groups, professionals and laypersons, with a system allowing choice of user group, and with two indexes, treating compounds in different ways. In accordance with the Cranfield Paradigm the MedEval test collection is based on three parts: A set of documents, a set of topics, and a set of known relevant documents with respect to each of the topics (Cleverdon, 1967). 1 The Document Collection The MedEval test collection is built on documents from the MedLex corpus (Kokkinakis, 2004). MedLex consists of scientific articles from medical journals, teaching material, guidelines, patient FAQs, health care information, etc. The set of documents used in MedEval is a snapshot of MedLex in October 2007, approximately 42 200 documents or 15.2 million tokens. See Table 1. For the MedEval test collection the documents are stored in the trectext format. The documents have IDs that reveal the source, and they are tokenized and tagged. Kristiina Jokinen and Eckhard Bick (Eds.) NODALIDA 2009 Conference Proceedings, pp. 223–226